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. , ... . Tl^ere is a continuing need in evaluation research for 

the establishment of a relationship between evaluation findings and ~ 
decision making, a method is, proposed for a particular situation: 
annual funding decisions for projects in a lar^e grant program in 
vocational education, outcome and predictive impact variables were 
ranked ty .three groups cf decision makers on a pilot study. The 
groups included the Director of the State Department of Education 
division responsible fcr funding decijgions, the supervisors who make 
funding decisions, and the supervisors from related bureaus who 
review aUd contribute to the decision making p):oc€ss. Statements 
concerning the impact of vocational educa^fion programs on students, ~ 
employers, and the State Department of Educlticn— to be used as 
program evaluation criteria— were sorted into twelve outcome impact 
and nine predictive impact statements. Each statement was ranked and 
rated for importance by the decision makers. Results showed high 
agreement on" the ranking and rating of outcome impact statements, a^d 
discrepancies on the pr-edictive impact statements. A validation study 
has been designed. Evaiuators can assist decision makers in 
Identifying important outcomes; and in the process, define the 
decision to be made, the timi when it is made, and the data required 
to link evaluation and decision making. (Author/JAG) 
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A continuing problem in evaluation research is establishing a relation- 
ship between evi^luation findings and decision making* The objective of the 
work described here is improve this relationship in one setting for v 
decision making: annual finding decisions for grant programs under the 

1 - ' 

VocaflLonal Education Act of 1976, The methdd is being developed in 'a 
pilot| pro ject with a state . education department (SED) where the^fuqding 
decisions are made. 

^ <^ 
The specific purposes o^ the pilot project are: l: to develop definitions 

of outc^e variables that will^ when combined^ identify 'a "high"' impact project^ 

that is^ project which is high on important outcome dimensions for stud'ents^ 



employers^ So: the granting agency; and 2. to relate these definitions to 
funding decisions • For purpose 2>> definitions of impact variables (ev?ri|iatiop 
outcome variables) are translated into "predictive impact" variables that 
canv be determined for each proposal at the time of the application for funding^ 

thus providing some "objectively determined" dati\to^be combined with the * 

\ 

other informatift^^ntering the fundibg decision making^^rocess* The develop- 
/ment of this method can improve the relationship between evaluation and 
, decision making by identifying the most important outcoDfes for program 

evaluation and decisiop making ;f or both project directors and SED decision 
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•makers,. By focusing attention on the aspects of these variables that can 
.fbe known at the time of funding, it il proposed that the likelihood of 

funding "high impact" projects will be increased, 

I * ' * 

Related Research ^ * . ^ - 

Davids and Salasin (19^5) have summarized man^-e4 the issues in the, 
u.se of evaluation results, including statements by evaluators that their 
results are not used, and those by administrators that evaluation findings 
are not available when decisions have to be made. Although there is much 
discussion of the need to relate evaluation^and decision making, there have 
been few efforts to specify thd manner in ^jfhich this* might occur, , 

Hemphill (1969) provided an early example of the use of educational 

evaluation d^ta in .a formal decision theory model. He illustrated several 

uses ot decision theory in evaluation^ In one ipstance, an in^lividual 

decision maker* decided whether to install a new nursery school, and also 
» 

^examined whether to -carry out the evaluation study, again vithin a decision 
theory framework. More recently, Edwards, Guttentag, atid Snapper (1975) 
have proposed and applied a'method called multi-attribute utility measurement 
(MAUM) to assist the Of f ice -tff Child Developmiant in defining the major 
dimensions of importance in d^eloping priorities for funding research projects, 

» The multi-attribute utility .^measurement method Is one of a set of me^thods 
classified ts^ decision aids by Slovic, Fischoff, and Lichtens^tein (1977), 
as opposed to formal. behavioral decision theory models, 

, Another decis4oij*='aid modej,- soqial judgment theory (SJT), has been 

described by Hammoad' and his* colleagues, and developed into an -infieractiya 
computer program. Both MAUM ^and> SJT are well-described in the literature 
(Edwards, 1976; Gardiner and^E^^wards, 1975 guttentag, and ^Snapper, 1975 ; Guttentag 
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1973; Gut ten tag. and Snappe^), 1974; Hammond and Adelman, 1^6; Hammond, 
Stewart^ Adelman, and Wascoe^ 1975; Hammond, Stewart, Brelimer and 
Steinmann, 1975). There is considerable overlap in the .^ast<^ ide^s of 
^the multi-attribute utility measurement method and «the,^QCiefi judgment 
theory as it is applied by Hammond. Ojie main difference, however, is in 
the nature of the judgment task presented to decision makers in order to 
determine the most important dimensions or vari^iSbles for decision making 
and their relative importance (say, for example, the i^ajc^r outcome variables 
of a program or the priorities among a set of goals Sox ,m funding program) 

The social judgment ^theorist, following Brunswik and the importance of , 

) ' 

representational validity, presents combinations of vaijiables as they 
would occur in real projects in the decisiorl setting xt orde.r to elicit 
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the utilities or values placed on the major variajales 
maker. The variables are judged (weighted) incj^pende 
d^eveloped by Edwards. / " ^ ^ 

The literature on decision aids is one area of riiivant research; 
the second area consists of defining "impact" for evai|iation purposes. 
Papers concerned with the analysis of impact and impa^ assessment method- 
ology have not always dealt wtth^the problem of how to define impact. ' 
However, representative definitions include, "the capacity of a program^to^ 
cause changes in those who are exposed to it" (Houston, 1972), and "the 
difference- between what' happens with the intervention and what would happen 
without" (Levine, 1976). Bernstein and Freeman(1975) deal with the definition 
problem by presenting the requirements for impact measurement: 

1. document the extent to which the social action program has 
or has not achieved its stated goals; 



, 2. attribute any effects or changes that are discovered to the 
.implementation of the* program; " i . 

3. delineate^ if possible^ the conditions or combinations of 

conditions under which the program is most effective;^ " 
4* delineate^ if possible^ any unan<;:icipated consequences or side ^ 
effects of the program. ' ' 

This definition assumes a common, set of go^ls across programs^ if one 
objective of . evaluation is to permit a comparison of the impact of 
different programs within a major funding program, Sirois and Iwanicki 
(1977) have noted the between-program comparison problem^ wliere program 
goals lack specificity and regularity. The present project is concerna^ 
with a somewhat different perspective^ since not all programs funded^ under 
the VEA can be expected to meet all the goals or priorities of the 
legislation/ . * 

' Hu and Stromsdorfer (1975) defined general criteria for measuring 

J ' ■ : " ■ 

the .educational and economic impact of research and demonstration projects - 

in vocational education • Two -types of impact of vocational education were 

identified: 1, intermediate impact or goals, "and 1. final output or ultimate 

goals. The first type included: modification or revision of curricula; re- 

allocation of funds within the educational system; effects on^students' 

aptitudes and school performance; number of graduates produced; percentage 

of graduates working in occupations for which they were prepared; improvement 

in student attendance; 'and sense of fulfillment in vbcational education 

teachers after developing a. new program,. The final* output included libor 

market performance of students (wages^ employment, job satisfaction) and 

W 

educational attainment, 

and Strtjmscforf er ' s list can be viewed as a general set of criteria. 

In addition tb the types of variables in*their list., the VEA for 1976 



provides that training for special populations is a^.so an important goal 
for VEA programs. The diversity of (legitimate) outcomes means that not 
all programs will have the same set of objectives. And, there is a concern 
expressed by stdte decision makers and local program administrators that 
not every program can meet the sme set of standards, when programs serve 
diverse populations, as in the vocational educatipn legislation, it is 
more realistic to have a set of "impact" variables to evaluate program 
impact, not all of which are expected to' apply to each ^program. Projects 
can be judged to have a high impact by meetings some standards (tha-t is, 
being high on some impact scales) but not on all. The same conclusion can 
be drawn for the predictive impact variables being identified in the 
current project. Where federal legislation has multiple goals and groups 
to be served, a set of important outcomes tl^^t are operatibnally defined 
may perinit identification of "high impact" projects and. also permit local 
needs.j:)|o be met. Yet, the definition of the the impact variables and the;ir 
use in^funding decision making may serve to focus local projects and evalua- 
tions on these same outcomfes, ^ ^ ^ , • ' 
Method^s and Results to Date 4 

The work>i:o date has generally followed the Edwards,^ Guttentag, and 
Snapper (1975) proceduire to develpp the impact dimensions. As noted by - 
both Hammond, Stewart, Brehemer and Steinmann(1975) and Edwards '(1976) the 
most important step is the first one; developing a clear understanding of 
the decision making process and developing the lists of variables to be 
considered for rating. ^Jititerviews were conAuoted Wth SED decision' makers 
tQ develop -a flow chart of the decision process for funciing, and to elicit 
statements defining. or critical to "high impact" projects^. In addition, 
•a review of the literature in vocational education was concjluctdd to identify 
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Other goals andj objectives • An initial lis't of 104 statements r.elated td 
impact of vocational education programs on students, employers, and the 
SED were sorted and reduced to 12 outcome (^pac.t) statements. These 
statements ^^jere rephrased to nine predictive impact statements by' identi- 

J / 

fying the variables that were known conditions to achieving the outcomes* 
(See Table 1 for sample statements*) Rank ordering and ratings of import- 
ance by decision makers were then obtained. At the time of the rating, 
all the dimensions were specified operationally, so raters" had an idea 
of what eventual scale definitions might be, even though on a tentative 
basis* - • 

Three rater "groups"' were used. The first "group" consisted of the 
Director of Dhe SED division responsible for funding decisions; trts second 
group was tlije set of supervisors who make funding decisions; and the third 
group was a set of supervisors from related bureaus who also review and 
Have a part in the decision making process. Agreement *amorgthe three 
"groups" of raters was, measured by Kendall's Coefficient of Concordance (W) • 
The coefficients for the twelve^ outcome statements were ,81 for the ranks 
'and ,94 for the ratings (£<;.01). Agreement was not a^ high for the 
predictive impact statements: W * ,.81 05 <£ <.J)r) for the ranks; and ' 
• W =» .41 (.50^<£<.30) for the ratings' of the importance of the predictive 
impact statement?. ' v 

The high agreement among the raters on the set of ratings and rankings 
of the outcome impact statements was* encouraging. It was not clear why 
there was a discrepancyv between the agreement 'on rank^s and that for the 
ratings for the predictive impact ^statements . As a result, there has been 
a revision of the two sets of statements, anciS^ second set of ratings and 
-rankings will be obtained. Some statements which were not ranked highly 



in either set of statements have been removed, and the two sets are 4iow par< 



Table 1 
Sample Outcome Impact 
Statements and Categories* 

. TRAINING OBJECTIVES ARE MET WITH MINmAL COST PER STUDENT > 

Training cost per student: 

$300 or less $504^. to $1000 $1001 to $1500 $1501 or more 

PROGRAM GRADUATES ARE WORKING IN OCCUPATIONS FOR WHICH THEY WERE ^TRAINED > 

Percentage of -program graduates employed in occupations for which trained 
(within first si;x months) : ' ^ . . ^ , ^ 

(S-257o ^ 26-50% 51^757o 76-100% . 

THE VOCATIONAL EDUCATION NEEDS OF ^SPECIAL GROUPS ARE MET - THE 
ECONOMICALLY DISADVANTAGED ♦ THE HANDICAPPED, AND PERSONS WITiTlIMITED 
ENGLISH-SPEAKING ABILITY > 

' . . * 

'Percentage of students trained who are from these. special groups: 

0-25% 7 26-50% 5;-757o ; 76-100%^" . 

STUDENTS ARE TRAINED FOR OCCUPATION^ -TRADITIONALIJ DOMINATED BY 
tHE OPPOSITE SEX : " ' T 

Percentage of students in prograrii who are trained, for occupations 
traditionally dominated by the opposite sex.^ 

Q-25% 26^50% 51-75% 76-100% _ 

a ' * 

Occupatifly^s in which the proportion of women is less than 38% 
EMPIOYERS A RE SATISFIED WITH GRADUATES OF THE PROHRAM 

a. Percentage of graduates that employers rate as satisfactory on ' 
entry level skills: 

0-25% "26-50% • , 51-75% 76-100% 




percentage of graduates that emglo^S^s retaiji or promote (for 
a two-year -period) : 

0-257o 26-50% , 51-75% 76-100% 



* Predictive Impact Statements are often the same, with the' exception that 
they are stated in the future tense (...will be ...). ' 



A sunnnary of the method to date includes these steps: 

^> 

^ !• interviews of decision makers and suirveys of the literature ^ 
to identify critical aspects of "high impact" projects; 

/ i 

2. Devieloping lists of outcome variables from 1*^ abovej 
' '3. "free" sorting of -statements by evaluation staff; >^ 

4, For a reduced list (twelve or fewer statements'^ state both 
as. outcome and predictive statements; develop operational 
^ definitions and sample scales for raters, . . ^ 

. 5, Obtain rankings and ratings of the preliminary set''* 
The ratings or value (utilities) attached to the dimensions can then be 
used in formal decision theory or a Bayesian decision theory approach 
(See Winkler^ 1972)- • ' . ' 

The nepct steps in the project are to develop forms which .can be us^d 
to* provide the data needed for funding decision's and for evaluation (the ^. 
predictive and QUtcome impact ^tatemen^is^ respectively)* These data provide 
the^ information necessary to change the sample categories given raters to 
categories based on distribution data* For example^ the sample categories 
for cost per student in Table 1 are fictional* In order to know whether 



^ There are^ obviously, any number of psychological scaling.methods available, 

as well as the methods used 'in formal behavioral decision theory to obtain 
.utilities (see Slovic, et al, 1977) • The objective is to obtain a reduced 
set of statements and then to obtain weights for the final set of statements 
However, it is not clear that other than unitary weights have great value* 
For example, Dawes (1973) and Hammond, Stewart, Adelman and Wascoe^s 
studies ' (1975) had results suggesting that with, multiple judges equal weight 
may result • • * • . 
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a project is high or low on cost per student, actual data must be obtained • 
^nd gtouped for cona^ar-able programs. 

Data will be obtained for past projects on the outcome statements. 
A sample of evaluatio\& will be judged as to- their overall 4evel of impact ^ 
by decision makers and "scored" on the outcome variables by the evaluators. 
The relationship between the 'two measures will be obt^ned as one evidence 
of the "validity" of the impact, dimensions. Also, ratings and rankings 
of the ,two sets of impact variables may be obtained from program directors 
at the Local EducatitJn Agencies (LEA's) . These data will provide another ^ 
perspective on the impact statements. Other sources ofSrel ated validity * 
data would be rankings 'of the variables by the State Advisory' Council to 
the VEA. For -the long term .study of the validity of the predtctive impact 
variables, there will need to be a follow-up rating of the overall level 
of impact^ for *grant applications that were given predictive impact ratings. 

There are limitations to the methods being proposed here. In the first 
place, much of the validation proposed is circular, as ^ovic et al. (1977) 
have noted for the studies of the other methods of aiding decisions. 
Second, the impact variables described here, while clearly important for ' 
evaluation, are only one part of the information used for funding decisions. 
Other areas which are rat^d in funding include the general •management plan 
for carrying out the project^ the proposed staff, and the project evaluation 
plan. At this stage^ it is hot known what weight these variables carry in 
decision making, or whether they are over-riding variables. Un-less standards 
are met in these other areas projects may not be funded, regardless of impact 
ratings. 
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Third, there are the political concerns of the toethod- On^ political 
aspect is the relationship between the evaluator and the administrative 
decision makers. The evaluator must be sensitive to staff nefeds in 
discussing and describing ^reas of work that are nor typically accessible 
to outsiders. Particularly important is the need for detailed knowle(Jge 
of the funding 'decision making process as it currently^ exists (and see^ 
Edwards, 1976, for similar caveats), 'a second political aspect is the 
relationship between decisi-bn makers and outside .groups, such as state, ' 
legislators and others with interest in the funding of local projects! 
From the evaluatorfe viewpoint, the development of well-defined criteria 
would appear to have benefits in providing a rationale for funding 
decisions. From a staff viewpoint, such a development may repre^senC a 
loss of "degrees of freedom" in their^ecision making. Knowledge pf the 
values or importance ratings given to the impact variables by other groups 
.such as LEA projec^t directors or local and state-level advisory councils \ 
might lessen this last concern. 

Summary ' , 

The significance of the mejthod being developed here is* in providing 
one way in which evaluation and funding decisions can be linked in programs 
that sre continuing and large scale. One characteristic of the programs 
funded under the VEA is that not every project will meet or be evaluated 
highly on all major impact 4iinensions. Evaluators can assist decision maker 
to establish the set of ' impact dimensions important for funding ^cisions. 
In the process, operational definitions of these variables ' 'are established 
for both evaluators and t)roject directors. Evaluation and decision making ^ 
can, in some cases, be more closely related ff evaluators clarify the^xact 
decisions to be made, who makes the decisions, timing of the decision, and 



the nature' of the data necessary to make the decision. In the VEA. new 
progr^s will not hafl. available past evaluation data at the time of funding, 
bijt the use of 'predicive impact data can, it is hypothesized, increase ' 
the likelihood of funding projects that will^ later be judged as higher 

i4 impact,' " s ^ ^ 

In this example, and similar settings, ,evaluators may find that 
devoting effort to clarifying fthe types cf decisions that can be made 
and the data that can be provided will increase the use of evaluation 
results oJer the long terin, \ - ^ * 
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